Launch gateway hardening: probes, retries, rate limits, forwarding deadlines - #868

…ry policy configurable

gateway-error retried 502/503/504 as a bundle, so during overload every 503
was retried 3 more times, multiplying offered load exactly when the fleet was
saturated. The default policy now retries connection failures and 502/504
only, and attempts/retryOn/perTryTimeout are values-configurable.
perTryTimeout stays unset by default because the single route also carries
streaming and long-lived requests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The sidecar had memory annotations only. On nodes at full CPU request
capacity an unrequested sidecar is starved during bootstrap, its postStart
hook hangs, and new gateway pods never become Ready, which turns scale-out
into negative capacity. Request 250m by default (configurable, no limit).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The sync http-forwarder posts to the local inference server without an
explicit aiohttp timeout, so the client default of total=300s applies.
Non-streaming generations that take longer than 5 minutes are cut off
with a 500 while the inference server keeps computing the response.

Add a timeout_seconds field to Forwarder and LoadForwarder (default
3600s), overridable per deployment via forwarder.sync.timeout_seconds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Reject zero, negative, non-finite, and non-numeric values when the
forwarder config is loaded, instead of letting them reach
aiohttp.ClientTimeout at request time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

GET /v1/async-tasks/{task_id} was a sync handler doing a blocking
result-backend read (S3 on AWS) in the shared anyio threadpool. Poll volume
scales with outstanding tasks, so a large backlog fills the pool and every
other sync route queues behind it. The handler is now async and dispatches
the blocking read through its own CapacityLimiter(40), isolating polls from
the rest of the threadpool.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…st use

DBManager eagerly built five engines per process (sync/async x RW/RO plus a
NullPool engine), so every gateway worker held idle pools for engines it
never uses; only the async pair is used on the API path. At 4 workers per
pod this multiplied idle Postgres connections across the fleet and made
scale-out storm the reader's connection limit. Engines are now created on
first use per kind; credential-expiry refresh disposes and rebuilds only the
kinds actually in use.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…nt clamps

Adds a values-gated (default off) EnvoyFilter installing
envoy.filters.http.local_ratelimit on the gateway sidecars, inbound:

- Per-pod token buckets for GET /v1/async-tasks/{task_id} and
  POST /v1/async-tasks. Overflow 429s at the proxy and never consumes a
  gateway worker, so overload degrades to fast rejections instead of
  queueing collapse. Per-pod semantics scale the fleet ceiling with the HPA.
- A throttledTenants values list clamps named callers on those routes,
  matching both accepted Authorization forms (Basic base64 prefix and
  Bearer), replacing hand-authored VirtualService fault injection during
  incidents.

Traffic matching no descriptor is unaffected (large default bucket,
always_consume_default_token_bucket: false).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…-open

Adds a Redis-backed per-user, per-route-class rate limiter enforced inside
verify_authentication, where identity is resolved, using the existing
aioredis pool. Fixed 1-second windows keyed on (route class, user_id);
rejections return 429 with Retry-After before the request reaches a handler
or the threadpool.

Disabled unless user_rate_limits is set in the service config. enforce:
false gives a log-only rollout mode that counts and logs would-be
throttles without rejecting. The limiter fails open on any Redis error or
if the check exceeds 100ms, so enforcement can never add an availability
dependency.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…tanding set)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… 1.15

Reverts the workload-port guess: Envoy config_dump on ml-training-new
(istiod 1.15.0) shows inbound|http|80 and no inbound|http|5000; with the
wrong name the VIRTUAL_HOST patch skips silently and no route or tenant
bucket ever fires (observed live: filter present, rate_limits absent,
zero 429s under burst). Same naming verified on Istio 1.30 locally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Envoy 1.23's local_ratelimit only consults route-level rate-limit policies;
virtual-host-level support (vh_rate_limits) arrived in 1.24+. On Istio 1.15
sidecars the VIRTUAL_HOST patch merged cleanly into config (verified in
config_dump) but the filter never produced descriptors, so no bucket ever
fired. Observed live on ml-training-new: filter and vhost rate_limits both
present, zero 429s, zero filter stats. Route-level actions work on both
1.23 and current Envoy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…running

asyncio.run() in the sync getter raises RuntimeError when called from a
thread with a running event loop (sync session access inside async code);
dispose the underlying sync engine directly in that case. Adds the DB-7
parametrized test covering both call contexts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Envoy header_value_match on 'Bearer <id>'/'Basic <b64>' was case-sensitive on
the whole value, but FastAPI accepts the scheme case-insensitively, so a
throttled caller sending 'bearer <id>' bypassed the clamp entirely (verified
live on training: canonical Bearer 429'd, lowercase bearer passed 12/12).
Match the scheme with a case-insensitive regex group and the credential
exactly (base64/token stays case-sensitive).

The 15/5 rps-per-pod buckets were anchored to the pre-hardening fleet's
degradation points (10.7 healthy / 18.3 knee rps/pod), which were symptoms
of the threadpool metastability this branch removes, not capacity. The
hardened gateway measured ~577 rps/pod of passing poll traffic at p99 36ms,
and sustained slow-poll overload now degrades to fast shedding rather than
collapse. Buckets become a runaway-storm ceiling (GET 100/pod burst 200,
POST 20/pod burst 40); per-tenant fairness remains the app-level limiter's
job.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The postmortem's RC2 established the incident's binding resource was node
NIC egress from multi-MB inline poll results (~500MB/s/pod knee). At
100 rps/pod the bucket admits ~530MB/s/pod worst-case, i.e. no byte
protection. 50 rps/pod bounds worst-case egress at ~265MB/s/pod while
still admitting ~15x baseline (fleet floor 1,500 rps ~= incident peak).
Revisit upward once MLI-8311 replaces inline results with presigned URLs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Emits model_engine.user_rate_limit.decision (dogstatsd counter) tagged
user_id/route_class/outcome on every limiter decision (allowed,
would_throttle, throttled, fail_open, breaker_open). outcome:allowed is
the per-tenant volume signal on the rate-limited routes; the throttle
outcomes are enforcement telemetry, including detection of the limiter
itself being inactive. Replaces the postmortem's proposed raw per-tenant
request-volume monitor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>